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Abstract: 

Inspection of thousands of student responses to computer-marked assessment questions has brought insight into the 
errors made by adult distance learners of science. Most of the questions analysed were in summative use and required 
students to construct their own response. Both of these things increased confidence in the reliability of the findings, as 
did the fact that similar errors were seen in different variants and in different questions. Questions on logarithms, 
graphs and gradient, differentiation and standard deviation were poorly answered. However, the most persistent 
errors, seen in questions designed to assess a range of different skills, were in rounding numerical values to an 
appropriate number of decimal places or significant figures and in working out the units of an answer. Other errors 
included incorrect precedence in calculations, giving symbols in an incorrect case, and problems with reciprocation, 
adding fractions, handling powers, performing unit conversions, and simplifying algebraic expressions. Possible 
reasons for the errors include careless slips, poor understanding of a method, and deeper conceptual 
misunderstandings. Many of the errors were similar to those reported by others, both at university and school level, 
and it appears that there are some very basic stumbling blocks that affect a wide range of students. It is suggested that 
increased use of discussion with students might bring further insight into the reasons for errors as well as increasing 
student understanding. 
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Introduction 

The 'maths problem', identified as a stumbling block for higher education students across a range of 
disciplines in science and engineering (e.g. Tariq et al, 2005; Engineering Council, 2000), is not new. 
Declining numeracy skills have been being reported since at least 1876 (Dearing, 1996) and 'it has 
always seemed that things were better 20 years ago' (Gill, 1999b, p.84). Although there is considerable 
variation between mathematical abilities of 15-year old students around the world (OECD, 2013), the 
maths problem at entry to university has been reported in many countries, for example Ireland (Gill 
and O'Donoghue, 2007) and Australia (Cuthbert and MacGillivray, 2002) as well as the UK (Croft et 
al, 2009). The longstanding and widespread nature of the problem mean that, whilst it may be 
attributed in part to differences in the school curriculum and to widening access to higher education 
(Williamson et al, 2003) and other changes such as over-reliance on calculators (Dearing, 1996), it is 
likely that it has its basis in more fundamental misconceptions. 

A number of authors talk about the importance of building mathematical skills and understanding on 
a firm foundation, with useful analogies including a 'house of cards' in which 'the bottom layer must 
be firmly and accurately constructed if [the cards] are to support the next layer up'. (Butterworth, 
1999, p. 298). Tobias (1978) highlights the sequential nature of mathematics in her description of the 
'dropped stitch' problem, in which schoolchildren and adults struggle with more advanced 
mathematics if, for example, 'the day they introduced fractions I had the measles' (p. 55). Thus work 
on schoolchildren's mathematical misunderstandings is likely to be highly relevant for adult 
education (Rees and Barr, 1984), especially since there is often only slow development of 
mathematical understanding during the teenage years (Kuchermann, 1981; Ryan and Williams, 2007). 
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Several projects have analysed the mathematical misunderstandings of cohorts of children, either by 
inspection of large numbers of responses to questions in national tests (Williams and Ryan, 2000) or 
by the use of specially designed questions (Ryan and Williams, 2007), sometimes supplemented by 
interviews (Hart et al, 1981). Rees and Barr (1984) identified threshold 'core of difficulty' problems 
and asked students to 'talk through' their solutions. At higher education level, many universities 
attempt to diagnose the mathematical difficulties of individual students on entry to their programmes 
(e.g. Appleby et al, 1997), whilst Leopold and Edgar (2008) and Tariq (2008) have looked 
systemmatically at the mathematical errors made by cohorts of students of chemistry and biosciences 
respectively. Lists of common errors have been produced (e.g. Schechter, 2009) and several authors 
have identied particular threshold topics e.g. algebra (Lins, 1992) or graphs (Gill, 1999a) which they 
believe correlate well with success in subsequent study. 

Context 

The work described in this paper offers particular insight because it is an attempt to identify the 
errors made by adult students of science, from all over the world and with widely varying ages and 
mathematical backgrounds. This means that blame for weak mathematics cannot be attributed to a 
particular educational system, whilst there is every incentive to find out as much as possible about 
where the misunderstandings lie. 

The students were studying a 10-credit (100 hours of study) Maths for Science module at the UK Open 
University (OU). This module is in line with most of the OU's undergraduate curriculum in that it has 
no entry qualifications, though students are advised to take it after the introductory science module 
Exploring science and in preparation for higher level study of science. The age distribution of the 322 
students on the presentation of Maths for Science that started in October 2012 is given in Figure 1. 
Seventy-two of the students (22%) had previous educational qualifications below those normally 
assumed on entry to UK universities and although the majority (239 students i.e. 74%) were studying 
in England, 32 (10%) were in Scotland and 33 (10%) were outside the UK. 



Figure 1. The age distribution of the 322 students on the October 2012 presentation of Maths for Science 

The edition of Maths for Science offered to students from October 2012 is a slightly modified version of 
a successful module which ran in four presentations per year from 2002-2012 and was studied by 
more than 12,000 students, of similarly varied background to that described above. 

The book on which the module is based (revised for the new version of the module as Jordan et al, 
2013) has the following chapters: 

1. Starting points [Numbers; Fractions; Powers, reciprocals and roots; Doing calculations in the 
right order], 

2. Measurement in science. 

3. Calculating in science. 

4. Unit conversions. 

5. Algebra. 
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6. Putting algebra to work. 

7. Graphs and gradient. 

8. Rate of change and differentiation. 

9. Angles and trigonometry. 

10. Logarithms. 

11. Probability and descriptive statistics. 

12. Statistical hypothesis testing. 


When it was first offered to students in 2002, Maths for Science was the first OU module to use 
summative and formative interactive computer-marked assignments (iCMAs), though many other 
modules now also use this method of assessment. The decision to take this approach was based on a 
wish to give students instantaneous feedback and an opportunity to act on that feedback while it was 
still fresh in their minds (Gibbs and Simpson, 2004-5) thus acting as a 'tutor at the student's elbow' 
(Ross et al, 2006 p.125), particularly important in a distance-learning environment. The e-assessment 
system used developed into the OpenMark system, many of whose features have now been 
incorporated into the Moodle quiz engine (Butcher, 2008; Hunt, 2012). Maths for Science has a 
summative iCMA-based end-of-module assessment (EMA), available to students in the final five 
weeks of the module, and a formative 'practice assessment' which students can access as many times 
as they want for the duration of the module. In both of these, students receive three tries at each of the 
questions, with an increasing amount of feedback provided for incorrect answers, as shown in 
Figure 2. A range of question types is used, with a preference for constructed-response, where 
students enter their own answer rather than selecting from options provided. Most questions exist in 
a number of variants, used to deter plagiarism in the summative EMA and to provide extra 
opportunities for practice in the practice assessment. 



Figure 2. A typical Maths for Science question with increasing feedback after subsequent tries (a)-(c) 

The teaching text of Maths for Science was originally developed in the light of known difficulties with 
mathematics in higher level science modules and from observation and discussion with students as 
they attempted mathematical problems. Misunderstandings revealed by an early analysis of some of 
the iCMA questions (Jordan, 2007) led to minor changes when the Maths for Science book was 
reprinted and also to improvements to the iCMA questions and feedback. For its entire life. Maths for 
Science has been shown to improve retention in subsequent science modules and it has been well 
received by students. However the more major rewriting of the book for 2012-13 provided the 
incentive to review systematically the performance of all the questions in the practice assessment and 
the two versions of the EMA which had been used on alternate presentations from 2002-2012. The 
questions were subsequently regrouped, with a small number of new items, into a single EMA for the 
revised module, an iCMA for another module, and a shared practice assessment. Other changes for 
the new edition of Maths for Science included a re-ordering of the material and the introduction of a 
series of screencasts explaining more challenging concepts. 
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Method 

Student responses to iCMA questions were analysed for several presentations of Maths for Science, 
including those that started in February 2010 (262 students) and September 2010 (285 students), which 
shared one EMA, and May 2010 (478 students) and November 2010 (415 students) which shared a 
different EMA. Where there were no significant differences between presentations, data from several 
pre-rewrite presentations were combined in order to increase the data-set and thus to increase the 
confidence in the results and conclusions. The analyses were repeated for the EMA for the 322 
students in the first (October 2012 start) presentation of the revised module. Results were compared 
with those for similar questions prior to the revision and are discussed when there was a significant 
change; it was not considered appropriate to combine pre- and post-revision data. However, for 
clarity in this paper, all results are stated in the order of the teaching in the post-revision module. 

For each question in each assignment, two analyses were conducted: 

• An analysis of the number of students who got the question right at the first try, second try, 
third try, or not at all. This was used to give a quick indication of question difficulty and the 
effectiveness of the feedback provided. 

• An inspection of the actual correct and incorrect responses given. This was used to investigate 
in more detail the errors that students made. 

Blank and repeated responses, associated with poor engagement with iCMAs and with weak 
understanding (Jordan, 2013), were removed prior to the second analysis. Although all questions in 
the EMAs and the practice assessment were analysed, most of the results reported in this paper are 
from responses to questions in the summative EMAs, because of a greater confidence that these 
responses were revealing understanding rather than guesswork. Similarly, constructed-response 
questions were generally found to be more revealing than selected-response question types such as 
multiple-choice, because responses constructed by students showed their true understanding rather 
than their selection of an option created by the question author. Finally, multiple variants of questions 
were particularly useful when different incorrect responses to the different variants could be 
attributed to the same mistake or misunderstanding. On some occasions specific errors could be 
identified more clearly by inspection of the patterns of responses given to each variant. 

It is important to distinguish between the error that a student makes and the underlying 
misconception (Hadjidemetriou and Williams, 2002), especially since some of the errors will 
inevitably have been as a result of simple slips (Ryan and Williams, 2007) and in other cases, the same 
incorrect response from different students may have been caused by different misconceptions 
(MacGregor and Stacey, 1994). Although some attempt will be made to speculate as to the possible 
underlying misconceptions held by Maths for Science students, the strength of the methodology is the 
unequivocal identification of common errors, especially given that the same errors were made from 
presentation to presentation and thus by very large numbers of students, who came from a wide 
range of different backgrounds and had then studied a common 10-credit module. 

Results 

Overview of the assignment 

Figure 3 shows the number of students who, for each question in the EMA for the presentation of 
Maths for Science that started in October 2012, got the question right at the first, second or third try, or 
not at all. These results are broadly in agreement with similar analyses for other presentations. It is 
clear that questions on Chapter 1 (arithmetic). Chapter 9 (angles and trigonometry) and Chapter 12 
(statistical hypothesis testing) were well answered. In general, students appear to be very good at 
following the processes required by a particular statistical test, such as a chi-squared test (Questions 
37-39) and interpreting the result (Questions 40-41). 

Students are often thought to have difficulties in rearranging equations, so it is of considerable 
interest that the questions on algebra (Chapter 5) were also reasonably well answered. However, 
questions such as Question 11 in Figure 3, which require students to substitute values into an 
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equation without any rearrangement, have persistently been amongst the most poorly answered. 
Jordan (2007) identified errors in questions of this type as being caused by difficulties in finding the 
correct units (often linked to difficulties with fractions and reciprocation), giving answers to 
inappropriate precision, and incorrect use of precedence in arithmetic. Question 11 is discussed 
further in the following sub-section as are other individual questions showing poor performance, 
including Question 20 (finding correct units again) and Question 36 (standard deviation). 



Chapter 

5 6 7 8 


10 11 


12 


ii 



15 20 25 

Question 


Ull 

30 



.•-■I 


I I Correct at first try 
I I Correct at second try 
Correct at third try 
IHI Incorrect after 3 tries 


Figure 3. The number of responses that were correct at first, second or third try or not at all, for each 
question in the EMA (October 2012 presentation) 

Chapters whose questions were less well answered overall included Chapter 7 (graphs and gradient) 
and Chapter 8 (elementary differentiation), whilst two of the three questions on logarithms (Chapter 
10) were poorly answered. Both of these questions were selected-response questions, illustrating that 
selected-response questions are not necessarily easier than constructed-response questions. It was not 
possible to draw any firm conclusions about the nature of students' misunderstandings from the 
incorrect responses given to these two questions - it seems likely that many students were out of their 
depth and so simply guessing. 

Analysis of responses to individual questions. 

The following subsections consider responses to individual questions, where it has been possible to 
identify the most common mistakes that were made. The question numbers given relate to the 
position of a question or a similar one in the EMA for the October 2012 presentation (as shown in 
Figure 3). Thus 'EMA Qll' describes the question that was Question 11 in this EMA, even though it 
was actually Question 10 in the earlier EMAs on which the analysis was carried out; 'EMA Qlla' 
describes a question that was similar to EMA Qll, but used on a different EMA whilst 'PA Qlla' 
describes a question that was similar to EMA Qll, but used in the practice assessment. Most of the 
questions exist in several (typically five) variants, but following work to ascertain that the variants 
were of equivalent difficulty (Jordan et al, 2012), the results are reported as if all for one variant. Thus, 
for example, EMA Q2 (the first question discussed in the following subsection) is described as 

3 2 

requiring students to add — + — , but actually roughly equal number of students received this variant 

2 5 

and four similar ones such as —I—. 

3 7 


3 2 

Chapter 1. Arithmetic fractions. Of 1233 responses to EMA Q2 ('What is — + —expressed as a single 

fraction? You should give your answer in the simplest possible form.'), 1098 (89.1%) were correct. 
Thus, as shown in Figure 3, this question was well answered. The common errors observed in 
responses to this question are given in Table 1, with the most frequent mistakes both involving the 
separate addition of numerators and denominators. 
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Table 1. Common errors in 1233 responses to EMA Q2 

Probable error 

Response 

Number of responses 
(percentage) 

Found common denominator then added 

19/30 

24 (1.9%) 

numerators and denominators separately 
Added numerators and denominators 

5/8 

16 (1.3%) 

separately 

Multiplied the fractions rather than adding 

6/15 

9 (0.7%) 


Of 1262 responses to EMA Q3 ('What is — -s-4 expressed as a single fraction? You should give your 

answer in the simplest possible form.'), 1106 (87.6%) were correct. However, as shown in Table 2, it is 
interesting that the most common error was to fail to give the answer in its simplest form. 


Table 2. Common errors in 1262 responses to EMA Q3 


Probable error 

Response 

Number of responses 
(percentage) 

Divided correctly but did not give answer in 
the simplest form 

2/12 

51 (4.0%) 

Multiplied the fractions rather than dividing 

8/3 

24 (1.9%) 


Chapter 1. Powers. Of 1259 responses to EMA Q4 ('Evaluate (3 10 ) 1 / 5 . You should give your answer as a 
single number.'), 1085 (86.2%) were correct. The most common errors are given in Table 3, illustrating 
that whilst the majority of students answered this question well, a significant minority confused 


multiplication and raising a number to a power. Where students found 



instead of (3 10 ) 1 / 5 , 


their precedence error was perhaps caused by poor calculator use. As shown in Table 4, similar errors 
were seen in 16476 responses to a question on the practice assessment, PA Q4a, 'Evaluate 1000 2 / 3 . You 
should give your answer as a single number.' 


Table 3. Common errors in 1259 responses to EMA Q4 


Probable error 

Response 

Number of responses 
(percentage) 

3x10x1 

6 

36 (2.9%) 

3 10 xl 

11809.8 

30 (2.4%) 

iox3 

3x10 5 =3xl0 2 

300 

14 (1.1%) 

Table 4. Common errors in 16476 responses to PA Q4a 

Probable error 

Response 

Number of responses 
(percentage) 

lOOOxf 

666.7 

770 (4.7%) 

1000 1 / 3 

10 

615 (3.7%) 

(1000 2 ) 4- 3 

333333.3 

226 (1.4%) 


For EMA Q4a, shown in Figure 4, 1090 (70.6%) of 1543 responses were entirely correct. However it is 
noteworthy that whilst 1400 responses (90.7%) included one of the correct options, 3 5 x3 2 , only 1174 

responses (76.1%) included the other correct option, 3 3 -^3 -4 , illustrating that negative powers 
notation and reciprocation cause difficulty for some students. The most commonly selected incorrect 
option was 3 4 + 3 3 , included in 217 responses (14.1%). 
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Which two of the following expressions are 
equivalent to 3 7 ? 

|nj 3 i4_ i _ 3 2 

□ 3 5 x3 2 

□ 3 3 3-4 

□ 3 14 x 3 1,2 

□ 34 33 



Figure 4. EMA Q4a 

Chapter 2. Decimal places, significant figures and rounding. Of 1363 responses to EMA Q6a ('Express 348.7 
in scientific notation to three significant figures.'), 871 (63.9%) were entirely correct. Most students 
handled scientific notation well, with 1181 responses (86.6%) being expressed in correct scientific 
notation, but significant figures and rounding caused more difficulty. The most common errors are 
given in Table 5. 


Table 5. Common errors in 1363 responses to EMA Q6a 


Error(s) (other aspects of the answer 
correct) 

Response 

Number of responses 
(percentage) 

Answer given to 4 sig figs (should be 3) 

3.487 x 102 

98 (7.2%) 

Answer truncated rather than rounded up 

3.48 x 102 

98 (7.2%) 

Answer not in scientific notation 

349 

64 (4.7%) 

Answer not in scientific notation and 
truncated rather than rounded up 

348 

17 (1.2%) 


It has been thought that students might find it more difficult to write a number in scientific notation 
where the power of 10 was negative, but EMA Q6 ('Express 0.8486 in scientific notation to three 
significant figures') was in general better answered than EMA Q6a, with 946 (68.9%) of 1374 
responses to EMA Q6 being completely correct. Although some responses gave the incorrect sign for 
the power of 10 (Table 6) it was apparently less tempting to fail to convert to scientific notation than 
was the case for EMA Q6a (Table 5). No responses of 0.848 (truncated and not in scientific notation) 
were observed. EMA Q6 also appeared to be slightly better answered in the presentation that started 
in October 2012, with 257 (71.4%) of 360 responses being entirely correct, and proportionally less of 
each of the errors, perhaps because of more careful teaching on rounding and the introduction of 
screencasts. 


Table 6. Common errors in 1374 responses to EMA Q6 


Error(s) (other aspects of the answer 
correct) 

Response 

Number of responses 
(percentage) 

Answer given to 4 sig figs (should be 3) 

8.486 x l0~i 

84 (6.1%) 

Answer truncated rather than rounded up 

8.48 x lO 1 

67 (4.9%) 

Incorrect sign on power of 10 

8.49 x 101 

34 (2.5%) 

Answer not in scientific notation 

0.849 

26 (1.9%) 


Decimal places, significant figures and rounding were found to cause problems throughout the EMA, 
so two simpler questions were introduced for the presentation that started in October 2012. The first 
of these, EMA Q8, is shown in Figure 5. Although some caution is necessary in interpreting the 
results, because of the relatively small number of responses to this question that were available for 
analysis, these responses bring further insight into where the difficulties lie. 261 (66.1%) of the 395 
responses to this question were completely correct and whilst 375 (94.9%) correctly identified 0.036 as 
being given to three decimal places and 342 (86.6%) correctly identified 0.036 as being stated to two 
significant figures, only 309 (78.2%) correctly identified the fact that 4.310 is stated to four significant 
figures. The most common errors are given in Table 7. 
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Enter a number into each of the four boxes 
in the table below to indicate the number 
of decimal places and the number of 
significant figures in each of the figures 
given. 



Number of 
decimal 
places 

Number of 
significant 
figures 

0.036 



4.310 




Figure 5. EMA Q8 


Table 7. Common errors in 395 responses to EMA Q8 
Error(s) (other aspects of the answer correct) Number of responses 
_ (percentage) _ 

4.310 identified as being to 3 sig figs 33 (8.4%) 

4.310 identified as being to 3 sig figs and 2 14 (3.5%) 

decimal places 

4.310 identified as being to 2 decimal places 13 (3.3%) 


Chapter 3. Calculating in science. EMA Q10, designed to assess students' ability to manipulate powers 
of 10, with or without a calculator, and their understanding of order of magnitude, is shown in Figure 
6. Of 1678 responses to this question, 968 (57.7%) gave the correct answer of 13. Table 8 reveals that 
most students used a calculator, not always correctly, and that the most common error was to give the 
power of 10 of the calculated value rather than rounding up to the correct order of magnitude. 


Estimate the value of the following 
expression. You should give your answer to 
the nearest order of magnitude. You may 
choose to do the calculation either with or 
without a calculator. 


IQ 1 - < Jsi * IQ 10 

9 4 x IQ 4 


10 


Figure 6. EMA Q10 


Table 8. Some of the common errors in 1678 responses to EMA Q10 


Probable error 

Response 

Number of responses 
(percentage) 

Calculated answer as 9.57 x 10 12 , then gave 
power of 10 rather than order of magnitude. 

12 

189 (11.3%) 

Entered 10 12 into calculator as 10E12 
(i.e. as 10 x 1012). 

14 

75 (4.5%) 

Found 10 12 xV8Ix10 10 -e(9.4x10 4 ) 

18 

52 (3.1%) 

Found 10E12xV8lxl0E10^(9.4xl0 4 ) 

20 

40 (2.4%) 

Found 10E12xV8lxl0E10^(9.4xl0 4 ) 

then gave power of 10 rather than OOM. 

19 

30 (1.8%) 


EMA Qll, EMA Qlla and PA Qlla, designed to test similar skills, are compared in Figure 7. 
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If V= 1.5 x lo -5 m 3 and 

r= 1.57 x io -2 m calculate h in the equation 



You should give your answer in scientific 
notation, with the correct number of 
significant figures and the correct SI base 
units. 


h = 

□ Superscript (t) 


If 1 = 0.65 m and g = 9.81 ms 2 , find T in the 
equation 



You should give your answer with the 
correct number of significant figures and 
the correct SI units. 

(N.B. You do not need to understand the 
underlying science in order to answer this 
question.) 

x‘ 

T = 


If 1 = 6.1 x io 30 W and 
F = 4J x 10' 10 Wm' 2 , 


find d in the equation d = 


nr 

J4nF 


You should give your answer in scientific 
notation, with the correct number of 
significant figures and the correct SI base 
units. 


(N.B. You do not need to understand the 
underlying science or the units used in 
order to answer this question.) 


d = 

□ Superscript (t) 


(a) 


(b) 


(c) 


Figure 7. (a) EMA Qll; (b) EMA Qlla; (c) PA Qlla 

None of these questions were well answered, with just 1220 (38.2%) of 3190 responses giving the 
correct answer to EMA Qll; 1039 (32.0%) of 3242 responses giving the correct answer to EMA Qlla; 
and 3111 (38.6%) of 8054 responses giving the correct answer to PA Qlla. As previously reported 
(Jordan, 2007), the errors were mostly as a result of problems with units and rounding to an 
appropriate number of significant figures. However, the different behaviour of the three questions 
provides additional insight. Common errors to EMA Qll, EMA Qlla and PA Qll are given in Tables 
9,10 and 11 respectively. 


Table 9. Common errors in 3190 responses to EMA Qll 


Error(s) (other aspects of the answer 
correct) 

Response 

Number of responses 
(percentage) 

Units given as m 2 (should be m) 

5.8 x 10 2 m 2 

205 (6.4%) 

Answer given to 3 sig figs (should be 2) 

5.81 x 10 2 m 

111 (3.5%) 

No units 

5.8 x 10 2 

80 (2.5%) 

Calculated 3 x V h- n x r 2 

3.5 x 10 ^ m 

77 (2.4%) 

Units given as m 3 

5.8 x 10~ 2 m 3 

49 (1.5%) 


Table 10. Common errors in 3242 responses to EMA Qlla 


Error(s) (other aspects of the answer 

Response 

Number of responses 

correct) 


(percentage) 

Answer given to 3 sig figs (should be 2) 

1.62 s 

336 (10.4%) 

Units given as s _1 (should be s) 

1.6 s -1 

169 (5.2%) 

Units given as s 2 

1.6 s~ 2 

156 (4.8%) 

Units given as m s 1 

1.6 m s n 

109 (3.4%) 

Answer given to 3 sig figs and units as s 

~ 2 1.62 s~ 2 

97 (3.0%) 

Answer given to 3 sig figs and units as s 

i 1.62 s -1 

94 (2.9%) 

Units given as m s 2 

1.6 m s~ 2 

85 (2.6%) 

Answer given to 3 sig figs and no units 

1.62 

80 (2.5%) 

No units 

1.6 

28 (0.9%) 

Table 11. Common errors in 

8054 responses to PA Qlla 

Error(s) (other aspects of the answer 

Response 

Number of responses 

correct) 


(percentage) 

No units 

3.2x10" 

1296 (16.1%) 

Units given as m 2 (should be m) 

3.2 x 10" nT 2 

698 (8.7%) 

Units given as m 2 

3.2 x 10" m 2 

571 (7.1%) 

Incorrect rounding? 

3.3 x 10" m 

435 (5.4%) 

Student has calculated d 2 rather than d 

1.0 x 10 39 m 2 

324 (4.0%) 

























72 | European Journal of Science and Mathematics Education Vol. 2, No. 2, 2014 


(usually with units of m 2 ) 

Answer given to 3 sig figs (should be 2) 3.21 x 10 11 m 

Units given as ml 1 3.2 x 10 u ml 1 

Units given as W nT 2 3.2 x 10 u W ml 2 


234 (2.9%) 
205 (2.5%) 
178 (2.2%) 


EMA Qlla was less well answered than the other two, even though this question was used in a 
summative EMA. Many responses included multiple errors, but at least 607 responses (18.7%) gave 
an answer to three significant figures rather than the expected two. Significant figure errors were seen 
in substantially fewer responses to EMA Qll and hardly at all in responses to PA Qll. This can be 
explained by the fact that all the input data for PA Qll was clearly given to two significant figures 
and, whilst r in EMA Qll was given to three significant figures, V was clearly given to two significant 
figures (which, according to the teaching in Maths for Science, means that the answer should be given 
to two significant figures too). In contrast, in EMA Qlla, g was given to three significant figures, and 
it appears that many students incorrectly thought that L = 0.65 m was to three significant figures too, 
which would mean that the answer should be given to three significant figures. 

Many responses to all three questions were incorrect because they gave incorrect units. For EMA 
Qlla and PA Qlla it is possible to further attribute these errors to problems with finding a reciprocal 
of a negative power (leading to units of s n instead of s), taking a square root (leading to units of m 2 
instead of m) or both (leading to units of s~ 2 instead of s and ml 2 instead of m). The fact that students 
found it more difficult to give their answer to EMA Qlla to an appropriate number of significant 
figures than was the case for the other two questions was partly compensated for by the fact that 
16.1% of the responses to PA Qlla, 2.5% of the responses to EMA Qll, but only 0.9% of the responses 
to EMA Qlla had omitted units. For PA Qlla, the omission may be attributed to students paying 
insufficient attention to a purely formative question, whilst for EMA Qll, some students probably 
thought that the correct answer was dimensionless as opposed to simply forgetting to give units. 

There was some evidence of errors with precedence in EMA Qll (students calculating 3 x V -h n x r 2 
instead of 3 x V + n + r 2 ). However these appear to have been calculator errors rather than being 
caused by a fundamental misunderstanding of the nature of the calculation, otherwise the units 
would have been given as m 5 , which was not seen. 

When EMA Qll was used in the presentation of Maths for Science that started in October 2012, it was 
significantly better answered than previously, with 212 (45.7%) of 464 responses being correct and 
proportionally less of each of the errors. 

Chapter 4. Unit conversions. The question used as EMA Q13 in the presentation that started in October 
2012 ('A jug has a volume of 1.01 x 10 3 cm 3 . Express this value in dm 3 .') had previously been used in 
the practice assessment, and not surprisingly it was better answered in the EMA (with 244 i.e. 63.2% 
of 386 responses being correct) than it had been as PA Q13 (when 5919 of 11551 i.e. 51.2% of responses 
were correct). On each occasion, by far the most common incorrect responses were those that were 
100 times too large, corresponding to division by 10 rather than by 10 3 i.e. to a conversion from cm to 
dm rather than from cm 3 to dm 3 . When used as EMA Q13, 41 responses (10.6%) included this error. 
The common errors made when the question was used as PA Q13 are given in Table 12. 


Table 12. Common errors in 11551 responses to PA Q13 


Error(s) (other aspects of the answer correct) 

Response 

Number of responses 
(percentage) 

Divided by 10 rather than 10 3 

1.01 x 10 2 

1219 (10.6%) 

Divided by 10 2 rather than 10 3 

1.01 x lOi 

897 (7.8%) 

Multiplied by 10 3 (conversion 'the wrong 
way round') 

1.01 x 106 

652 (5.6%) 

Multiplied by 10 (conversion 'the wrong way 
round' and linear rather than cubed) 

1.01 x 1Q4 

408 (3.5%) 
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EMA Q13a ('The area of a cricket pitch is 2.6 x 10 2 km 2 . Express this value in scientific notation in 
m 2 .') showed similar errors (Table 13) to those seen in PA Q13. 


Table 13. Common errors in 1582 responses to EMA Q13a 


Error(s) (other aspects of the answer correct) 

Response 

Number of responses 
(percentage) 

Multiplied by 10 3 rather than 10 6 (i.e. forgot 
to square the conversion factor) 

2.6 x 10 1 

222 (14.0%) 

Multiplied by 10 4 rather than 10 6 

2.6 x 10 2 

62 (3.9%) 

Divided by 10 3 instead of multiplying by 10 6 
(conversion 'the wrong way round' and 
linear rather than squared) 

2.6 x 1(T 5 

37 (2.3%) 

Multiplied by 10 5 rather than 10 6 

2.6 x 10 3 

36 (2.3%) 

Answer not in scientific notation 

26000 

25 (1.6%) 

Divided by 10 6 instead of multiplying 
(conversion 'the wrong way round') 

2.6 x l(T« 

22 (1.4%) 


Of 1582 responses to EMA Q13a, 977 (61.8%) were correct, but 222 (14.0%) had not squared the 
conversion factor and 59 (3.7%) had done the calculation 'the wrong way round', sometimes 
combining this with failing to square the conversion factor. Some of the other errors may have been 
caused by misreading from a calculator display, especially since some students clearly did not work 
in scientific notation. Sadly, there was also evidence of students not checking that their answer was 
reasonable - playing a game of cricket on a pitch of area 2.6 x 1CT 5 m 2 or 2.6 x 1CT 8 m 2 might be 
difficult! 

EMA Q14 ('A pyroclastic flow from an erupting volcano travels at a speed of 65 km hour"' 1 . Express 
this value in m s' 1 to an appropriate number of significant figures.') provides an example of a 
question whose behaviour was so unexpected that the answer matching was altered prior to the 
October 2012 presentation. Before this time, only 945 (54.8%) of 1723 responses were marked as 
correct, but as Table 14 illustrates, most of the errors were with significant figures rather than with the 
unit conversion itself. Therefore, for the October 2012 presentation, it was decided to mark as correct 
responses that were numerically correct but given to more than two significant figures, in addition to 
those that were given to two significant figures and would have been marked as correct all along. Of 
the 354 responses received for this presentation, 49 (13.8%) were numerically correct but given to 
more than two significant figures. When these were included, a total of 260 responses (73.4%) of 
responses were now marked as correct. The other common errors, including forgetting to convert 
hour’ 1 to s 1 and performing this part of the unit conversion in the wrong direction, occurred with 
similar frequency to that observed for earlier presentations. 


Table 14. Common errors in 1723 responses to EMA Q14 


Error(s) (other aspects of the answer correct) 

Response 

Number of responses 
(percentage) 

Answer given to too many significant figures 

18.1 

290 (16.8%) 

Converted km to m but did not convert 
hour -1 to s 1 

6.5 x 10 4 

42 (2.4%) 

Out by a factor of 10 

1.8 

34 (2.0%) 

Multiplied by 3600 rather than dividing by 
3600 

2.3 x IQ 8 

26 (1.5%) 


Chapter 5. Algebra. Chapter 5 of Maths for Science includes sections on 'Rearranging equations', 
'Simplifying equations' and 'Combining equations' and Figure 8 shows three questions, each 
associated with one of these subsections. EMA Q15 (shown in Figure 8(a)) is a simple 'rearranging 
equations' question and 1039 (80.4%) of 1293 responses were correct (and, depressingly, as shown in 
Table 15, the most common error was to give the incorrect case for the symbols). However, although 
most of the 'rearranging equations' questions in the EMAs and the practice assessment are of 
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necessity selected-response question types (because of the difficulty for students to enter complex 
algebraic expressions) and therefore not very useful in revealing common misunderstandings, these 
questions were always relatively well answered, with 75% or more of responses being correct. 


Rearrange / = r- to make h the subject of 
n 

the equation. 

Give your answer in the simplest form and 
without brackets. Use the '/' key to format 
your answer, as appropriate. 



Which two of the following expressions are 
equivalent? 

Click on the boxes to select or deselect the 
options. 

E g+ 3 B a-+ 9 


Combine the following two equations to 
give an equation for a which does not 
involve e. 

You should give your answer in the 
simplest possible form and without 
brackets. 


B - B 

b fb 


Equation 1 

>»/ 
a =* —— 


(!) 1/2 


□ (a + 3)2 
(* + 3)2 


Equation 2 
/ “ ed- 


a = 

□ Superscript (t) 


(a) 


(b) 


(c) 


Figure 8. (a) EMA Q15; (b) EMA Q17a; (c) EMA Q18 


Table 15. Common errors in 1293 responses to EMA Q15 


Probable error 

Response 

Number of responses 
(percentage) 

E or f given with incorrect case (should 

e/for E/F 

79 (6.1%) 

be E/ f or E F 1 ) 



E multiplied by f rather than divided 

fE or Ef 

50 (3.9%) 

Reciprocal of correct answer 

f/EorfET 

30 (2.3%) 


For EMA Q17a, shown in Figure 8(b), 955 (59.1%) of 1615 responses had correctly selected the two 

\fa 


equivalent expressions, —^=and 


f \ 1/2 

a 




, (a + 3) 2 a 2 +9 

together in 275 (17.0%) of responses, were —-—— and 


. The most commonly selected incorrect expressions, seen 
, revealing a misunderstanding that 


(b + 3Y 


b 2 + 9 


expressions of the form (x + y)“ and X + V~ are equivalent. 


For EMA Q18, shown in Figure 8(c), 913 (63.7%) of 1434 responses were correct i.e. of the form md 2 or 
equivalent, and some common errors are shown in Table 16. Several of the responses indicate that 
students had difficulty in simplifying their answer. In particular, it appears that a substantial number 

mf 

of students got an equation of the form a = > which is equivalent to the correct answer even 

though not reached by the simplest method, but which requires further simplification. This 
simplification caused problems for many, with one apparent cause being that students did not 
appreciate that the'/ d 2 ' in the fraction's denominator is equivalent to d 2 in the numerator. 


Table 16. Common errors in 1434 responses to EMA Q18 
Probable error Response Number of responses 


(percentage) 


Incorrect simplification (error in 
handling of / d 2 in the denominator) 

m/ d 2 

131 (9.1%) 

Incorrect simplification 

mf 2 / d 2 or 
md 2 / f 2 

46 (3.2%) 

Insufficient simplification 

mf/f/d 2 

28 (2.0%) 

Insufficient simplification 

mfd 2 /f 

24 (1.7%) 
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Incorrect simplification (f should mfd 2 23 (1.6%) 

have cancelled) 


Chapter 6. Putting algebra to work. EMA Q20 (Figure 9(a)) requires students to work out the base units 
for a quantity from a (hypothetical though not unreasonable) equation. This question was added, 
along with a screencast on 'working out the units', following students' poor performance when asked 
to work out units in EMA Qll, and similarly poor performance in EMA Q20a (Figure 9(b)), which has 
been in use for a number of years. 


You have been given the equation a - 

a 

If b is measured in newtons (N), c in seconds (*) 
and d in metres (tn), what are the SI base units 
of a? You should use the standard abbreviation 
for the simplest form of the units. 


The SI base units of a are | 

Superscript (i) 


You have been given the equation 



If a Is measured In Joules and e Is measured In 
m »■*, what are the SI base units of b? You 
should use the standard abbreviation for the 
simplest form of the units. 


The SI base units of b are 


(•) (b) 


Figure 9. (a) EMA Q20; (b) EMA Q20a 

The performance of EMA Q20 for the October 2012 presentation is better than that of EMA Q20a for 
earlier presentations, with 210 (47.4%) of 443 responses to EMA Q20 in the October 2012 presentation 
being correct compared with 701 (33.7%) of 2079 responses to EMA Q20a for earlier presentations. It 
is not possible to tell whether this improvement is as a result of improved teaching or whether EMA 
Q20a is a simpler question than EMA Q20. Whatever, there is still scope for improvement here. The 
most common errors seen in responses to EMA Q20 and EMA Q20a are given in Table 17 and Table 
18 respectively. 


Table 17. Common errors in 443 responses to EMA Q20 


Probable error 

Response 

Number of responses 
(percentage) 

Correct units but not base 
units 

N s 2 nT 1 

44 (9.9%) 

Base unit equivalent of the 
newton (the unit of b) 

kg m s _2 

19 (4.3%) 

Capital K (i.e. Kg not kg) 

Kg 

8 (1.8%) 

Table 18. Common errors 

in 2079 responses to EMA Q20a 

Probable error 

Response 

Number of responses 
(percentage) 

Base units for a/e not a/e 2 

kg m s _1 

224 (10.8%) 

Base unit equivalent of the 
joule (the unit of a) 

kg m 2 s~ 2 

61 (2.9%) 

Unknown 

kg s 1 

50 (2.4%) 

Unknown (base unit 
equivalent of the newton) 

kg m s 2 

42 (2.0%) 

Unknown 

J ms _1 

41 (2.0%) 

Correct units but not base 
units 

J nT 2 s 2 

39 (1.9%) 

Capital K (i.e. Kg not kg) 

Kg 

35 (1.7%) 


Chapter 7. Graphs and gradient. EMA Q21 (Figure 10(a)) was correctly answered in 968 (67.7%) of 1430 
responses. A range of numerical values were accepted as correct and units were accepted as m/s or 
m s TThe commonly observed errors are given in Table 19, though it should be noted that many 
responses contained multiple errors, so the percentages given for each error are almost certainly 
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underestimates. The most common error was to calculate the gradient as if the line went through the 
origin, perhaps because students were just calculating distance/time, revealing a lack of 
understanding of the nature of gradient. 



Figure 10. (a) EMA Q21; (b) EMA Q21a 


Table 19. Common errors in 1430 responses to EMA Q21 


Probable error 

Response 

Number of responses 
(percentage) 

Gradient calculated as if line 
went through the origin 

4.2 m s -1 

85 (5.9%) 

Incorrect units 

3.3 m s 

76 (5.3%) 

Incorrect significant figures 

3.33 m s -1 

57 (4.0%) 

No units 

3.3 

36 (2.5%) 


EMA Q21a (Figure 10(b)) was correctly answered in 941 (59.9%) of 1571 responses. The most common 
errors (Table 20) again illustrate a lack of understanding of the concept of gradient, in this case with 
many students incorrectly believing the gradient to be negative because the values plotted (in this 
case for temperature in °C) are negative. 


Table 20. Common errors in 1571 responses to EMA Q21a 


Probable error 

Response 

Number of responses 
(percentage) 

Negative equivalent of 

-1.6 

264 (16.8%) 

correct answer 



Reciprocal of correct answer 

0.63 

42 (2.7%) 

Negative reciprocal of 

-0.63 

32 (2.0%) 

correct answer 




EMA Q23, shown in Figure 11, was designed to test students' understanding of the relationship 
between straight-line graphs and their equations. Both correct options were selected in 880 (57.4%) of 
1532 responses, though unsurprisingly (since the proportionality follows directly from the equation 
given in the question) more students appreciated that a graph of F m against 1/ r would be a straight 
line going through the origin than appreciated that r against 1/ F m would be a straight line going 
through the origin (Table 21). 
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J’u, and r are related hy the equation 

Kl-L 

F m = —— where all the other quantities In 
the equation are constant, which two of 
the following graphs would be straight 
linos going through llw origin: 

n ^ m agmnsn 

3 7 against j**,^ 

r l ,1 

— against - 


E 1 agasut d- 
r n> 



Figure 11. EMA Q23 

Table 21. Number of responses, from a total of 1532 for EMA Q23, which included each option 


Option 

Number of responses which 
included this option (percentage) 

F m against 1 /r (correct) 

1112 (72.6%) 

r against 1/ F m (correct) 

1028 (67.1%) 

F m against r 

326 (21.3%) 

1/ F m against 1/r 

256 (16.7%) 

F m against r 2 

181 (11.8%) 

r against F m 2 

136 (8.9%) 


It is interesting that the most commonly chosen incorrect response to EMA Q23 was that a graph of 
F m against r would be a straight line going through the origin. However, this question illustrates the 
difficulty of trying to gain insight into student misunderstandings from selected-response questions; 
it seems likely that some students were guessing whilst others were encouraged to select options by 
their presence in the question. It is difficult to relate the options chosen to students' true 
understanding. 


Chapter 8. Differentiation. EMA Q24 (Figure 12) is, like EMA Q23, a selected response question. 
However on this occasion, the pattern of option choice gives a clear indication of the nature of 
students' misunderstandings. The number of times that each option was selected is given in Table 22, 
but it is the combination of options that gives the clearest insight. Of 1438 responses to this question, 

d y 

868 (60.4%) correctly identified the three points at which —— = 0 as B, D and G, but this was followed 

dx 

by 312 (21.7%) responses that all gave the options A, C and E, with no other combination being seen 
in more than 3% of responses. Points A, C and E are the points at which y itself is equal to zero, as 

d y 

opposed to — being zero. A substantial minority of students are thus confusing a point on a graph 

dx 

with the gradient at that point. 



Figure 12. EMA Q24 
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Table 22. Number of responses, from a total of 1438 for EMA 24, which i ncluded each option 
Option Number of responses which 
_ included this option (percentage) 


D (correct) 

1051 (73.1%) 

B (correct) 

1046 (72.7%) 

G (correct) 

963 (67.0%) 

A 

443 (30.8%) 

E 

395 (27.5%) 

C 

377 (26.2%) 

F 

103 (7.2%) 


Of 1434 responses to EMA Q26 ('If z = 3 V + 5t + 6, what is the gradient of a graph of z against t at 
t — 2.'), 683 (47.6%) gave the correct response. However, the same fundamental error that was seen in 

dz 

EMA Q24 is seen in the 95 (6.6%) responses that evaluated z rather than — at t = 2, as shown in 

d t 


Table 23. 


Table 23. Common errors in 1434 responses to EMA Q26 


Probable error 

Response 

Number of responses 
(percentage) 

dz 

Evaluated z rather than — 

d t 

28 

95 (6.6%) 

dz 

Coefficient of first term of — (error more 

d t 

6 

76 (5.3%) 

clearly identified in other variants) 




Chapter 9. Angles and trigonometry. Of 1414 responses to EMA Q28a ('Convert an angle of 3.8° to 
radians. Give your answer to two significant figures.'), 997 (70.5%) gave the correct answer of 0.066. 
The most common errors are given in Table 24, illustrating the persistence of errors with significant 
figures and rounding. 


Table 24. Common errors in 1414 responses to EMA Q28a 


Error(s) (other aspects of the answer correct) 

Response 

Number of responses 
(percentage) 

Answer to 2 decimal places not 2 sig figs 

0.07 

149 (10.5%) 

Either incorrect rounding or approximate value 
used for n (answer close but not correct) 

0.067 

34 (2.4%) 

Answer to more than 2 sig figs 

0.0663 

11 (0.8%) 


Of 1491 responses to EMA Q29a (Figure 13), 939 (63.0%) gave the correct answer of 39 metres. The 
most common errors are given in Table 25. Students who calculated rfxtand instead of d/ tan0 may 
have incorrectly rearranged the equation tan(9 = d/h or they may have calculated the tangent of the 
incorrect angle. If the former, it is interesting that the principle errors are not with trigonometry but 
with algebra and rounding. 
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Figure 13. EMA Q29a 


Table 25. Common errors in 1491 responses to EMA Q29a 


Error(s) (other aspects of the 
answer correct) 

Response 

Number of responses 
(percentage) 

Calculated dtand instead of df tan0 

11 

122 (8.2%) 

Rounding error 

40 

93 (6.2%) 

Answer to more than 2 sig figs 

39.49 

31 (2.1%) 


EMA Q28a and EMA Q29a were not used the October 2012 presentation of Maths for Science, so it is 
not possible to assess the impact of the improved teaching, seen to reduce errors with significant 
figures and rounding in some earlier questions. 


Chapter 11. Probability and statistics. Of 1315 responses to Q35 (Figure 14), 962 (73.2%) gave the correct 
answer of 1/78. The most common errors are given in Table 26, with the most frequent being to use 
the addition rule for probability rather than the multiplication rule. 


A game starts by drawing one card from a 
shuffled pack of playing cards and 
simultaneously throwing a die. The highest 
possible score corresponds to drawing an 
ace and throwing a six. What is the 
probability, expressed as the simplest 
possible fraction, of achieving this highest 
possible score? Note that in a pack of 52 
playing cards, there are four aces. 


Probability = 


□ 

□ 


Figure 14. EMA Q35 


Table 26. Common errors in 1315 responses to EMA Q35 


Probable error 


Response 

Number of responses 





(percentage) 

Added fractions rather than 

19 

81 (6.2%) 

multiplying 


78 


1 


1 

1 

66 (5.0%) 

Used —rather than 



52 


13 

312 


Added numerators and denominators 

5 

15 (1.1%) 

separately 

' 4 r 
v 52 + 6; 


58 
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The analysis of responses was based on a slightly differently worded version of EMA Q35 from that 
shown in Figure 14. The earlier version did not explain that there are four aces in a pack of playing 
cards but instead referred students to the Maths for Science book for the description of a park of cards. 
It therefore seems likely that students who used 1/52 rather than 1/13 had misunderstood what the 
question was asking rather than making a mistake with the mathematics. The revised wording of the 
question was introduced prior to the October 2012 presentation and this resulted in substantial 
reduction of errors of this sort, with just one (0.3%) of the 342 responses using 1/52 rather than 1/13. 
However the question's overall success rate was similar, with 253 (74.0%) of responses being correct, 
and the other errors occurring in similar proportions. It is interesting that some students attempted to 
add the fractions rather than multiplying, but then added numerators and denominators separately. 
This error was seen in much earlier questions (e.g. EMA Q2) but given that a double error is required 
for this mistake to be seen in EMA Q35, it was in a sense a more common mistake here. 

Maths for Science teaches students how to calculate the standard deviation, a, of a series of repeated 
measurements, and this is assessed in EMA Q36 (Figure 15(a)). The module also teaches students how 
to use measurements from a sample to find the estimated standard deviation, s, of a larger 
population, and this is assessed in PA Q36 (Figure 15(b). Both questions were very poorly answered, 
with just 814 (49.7%) of 1637 responses to EMA Q36 giving the correct answer of a= 0.0420 and just 
1883 (37.4%) of 5037 responses giving the correct answer of s = 2.83. 

The common errors in EMA Q36 and PA Q36 are given in Table 26 and 27 respectively. Students 
appeared to confuse the two different standard deviations, to make errors in rounding to the required 
number of decimal places, or to make both of these errors. In the rewrite prior to the October 2012 
presentation, an attempt was made to improve the distinction between the two types of standard 
deviation, but this did not lead to an improvement in the accuracy of responses to EMA Q36. Of 447 
responses, 210 (47.0%) had correctly calculated a whilst 59 (13.2%) had calculated s. 


A length Is measured 10 times. The results are 
listed below. Calculate the standard deviation 
of these measurements to four decimal places, 
giving your answer as a decimal number (i.e. 
not in scientific notation). It may help you to 
know that the mean of these data Is 3 3839 m. 


A scientist studying Robins In Milton Keynes 
measures the wing lengths of 10 Robins 
selected at random. The results are listed 
below. On the basis of these data, calculate an 
estimated standard deviation of the population 
of Robins in Milton Keynes, giving your answer 
to two decimal places. It may help you to know 
that the mean of these data Is 74.35 mm. 

Reading 

Length/in 

Reading 

Leugtli/m 


1 

3.462 

6 

3.357 





2 

3.406 

7 

3.403 

Reading 

Wing 

length/mm 

Reading 

Wing 

length/nnu 

3 

3.452 

8 

3.355 

1 

76 0 

6 

75 5 

4 

3.359 

9 

3.349 


74.0 

7 

74.5 

5 

3.355 

10 

3.341 

3 

68.0 

8 

78.0 

Standard d 




4 

72 5 

9 

74 0 



5 

77 5 

10 

73 5 


estimated standard deviation * | | mm 


w (b) 


Figure 15. (a) EMA Q36; (b) PA Q36 


Table 26. Common errors in 1637 responses to EMA Q36 


Probable error 

Response 

Number of responses 
(percentage) 

Calculated s not a 

0.0443 

165 (10.1%) 

3 decimal places not 4 

0.042 

49 (3.0%) 

4 sig figs not 4 decimal places 

0.04203 

37 (2.3%) 

Calculated s not <7 and gave answer 
to 4 sig figs not 4 decimal places 

0.04430 

24 (1.5%) 

Calculated s not crand gave answer 
to 3 decimal places not 4 

0.044 

19 (1.2%) 
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Table 27. Common errors in 5037 responses to PA Q36 


Probable error 

Response 

Number of responses 
(percentage) 

Calculated cr not s 

2.68 

796 (15.8%) 

2 sig figs not 2 decimal places 

2.8 

220 (4.4%) 

Calculated cr not s and gave answer 

2.7 

148 (2.9%) 

to 2 sig figs not 2 decimal places 
Answer truncated not rounded up 

2.82 

118 (2.3%) 


Chapter 12. Statistical hypothesis testing. The final five questions in each of the assessments take a step- 
by-step approach through a different statistical test, for example, EMA Q37a-Q41a work through a 
two-tailed f-test for unmatched samples. In EMA Q37a (Figure 16(a)), students are required to 
calculate the common population variance and in EMA Q38a (Figure 16(b)) they use the correct value 
for this (which is stated at the beginning of EMA Q38a, so this question is not visible until EMA Q37a 
has been completed, and similarly throughout this section) to calculate the standard error of the 
differences in the sample means and so to find the test statistic t. 

In EMA Q39a (not shown), students find the number of degrees of freedom for the f-test. EMA Q40a 
(Figure 16(c)) then requires them to use a table of critical values to find the probability of obtaining a 
test statistic of this size by chance if the null hypothesis is true. The final question (EMA Q41a) offers a 
single attempt at a question with just two options: is there a statistically significant effect (at the 0.05 
level) or not? 


Of 1256 responses to EMA Q37a, 839 (66.8%) gave the correct answer of 3.546. The most common 
errors are given in Table 28, with the most frequent being to give the square root of the required value 


i.e. to find 



The mean number of flowers per plant (x), 
estimated population standard deviation of 
number of flowers per plant (s) and sample 
size (n) for each of the samples is given in the 
Table. 


Use the correct value of Sg = 3.546 and any 
additional data you require from the Table to 
calculate SEd (the standard error of the 
differences in the sample means) and hence to 
find the test statistic t. 


The absolute value of r is 0.913 (i.e. 
ignoring any minus sign), with 27 degrees 
of freedom. Select one option from those 
given below for the corresponding 
significance level. 


Variable 

North- 
West slope 
(sample 1) 

South-East 
slope (sample 
2) 

X 

6.56 

7.25 

s 

1.74 

1.94 

n 

9 

20 


Calculate Sg (the common population variance) 
for these data (to at least three decimal 
places). 



Variable 

North- 
West slope 
(sample 1) 

South-East 
slope (sample 
2) 

X 

6.56 

7.25 

s 

1.74 

1.94 

n 

9 

20 


You should enter your values for SEd and t to 
at least three decimal places. Note that the 
marks for this question will be awarded solely 
on the basis of whether the value you give for t 
is correct. However, if the value you give for t 
is incorrect, your answer for SEd will be used to 
check whether or not you are on the right 
lines. 


You will be allowed a maximum of two 
attempts at this question. 

Significance level 

O-P > o.i 

OO.l >P> 0.05 
00.05 > P > 0.01 
QP < 0.01 



t 


(a) 


(b) 


(c) 


Figure 16. (a) EMA Q37a; (b) EMA Q38a; (c) EMA Q40a 
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Table 28. Common errors in 1256 responses to EM A Q37a 


Probable error 

Response 

Number of responses 
(percentage) 

fF c not Si 

1.881 

78 (6.2%) 

Answer truncated not rounded up 

3.545 

31 (2.5%) 

3 sig figs not 3 decimal places 

3.55 

27 (2.1%) 


Of 1153 responses to Q38a, 843 (73.1%) gave a correct answer for f, with a further 65 responses (5.6%) 
giving a correct value for S E r>, but making an error in the calculation of t. The most common errors 
are given in Table 29. The appearance of a precedence (or calculator) error at this stage of the EMA is 


interesting, with students finding Xj 


X, X, - Xt 

-instead of-—. 

se d se d 


Table 29. Common errors in 1153 responses to EMA Q38a 
Probable error Number of responses 

(percentage) 

Assumed given value was for S c not fy , so squared again 

- X, . X, - X, 

X. -— mstead of-- 

se d se d 


21 ( 1 . 8 %) 
15 (1.3%) 


EMA Q39a is a selected-response (multiple-choice) question and the number and proportion of 
responses which selected each option are shown in Table 30. 

The critical values of f for 27 degrees of freedom are given in the table of critical values as 1.703 for 
P = 0.1, 2.052 for P = 0.05, and 2.771 for P = 0.01, thus 78.4% of responses were right to conclude that 
the calculated value for f of 0.913 implies that P > 0.1. However, since the critical values increase as 
the probability decreases, it is perhaps not surprising that some students interpreted the table as 
implying that P < 0.1 and so selected the next most popular option, 0.1 > P > 0.05. Since there is a 
relatively high probability of the result occuring by chance, the null hypothesis cannot be rejected and 
so the correct answer to EMA Q41 is that there is not a statistically significant difference, in this case 
between the mean numbers of flowers on the north-west and south-east slopes of a ridge. Of 905 
responses, 782 (86.4%) gave the correct answer. 

Table 30. Number of responses, from a total of 1081 for EMA 40a, which selected each option 


Option 

Number of responses which had 
selected this option (percentage) 

P > 0.1 (correct) 

848 (78.4%) 

0.1 > P > 0.05 

120 (11.1%) 

0.05 > P > 0.01 

60 (5.6%) 

P < 0.01 

53 (4.9%) 


Discussion and conclusions 

Errors seen in the Maths for Science questions include many that have been reported by others so, for 
example, students' difficulties with logarithms have been reported by Tariq (2003) and Leopold and 
Edgar (2008), whilst difficulties with basic calculus are as reported by Baker et al (1973) and Hoban et 
al (2013). 

Many of the observed errors are, as expected, in line with those made by schoolchildren, for example 
separately adding the numerator and denominators of fractions (Hart, 1981a; Swan, 2001) and 
confusing multiplication with raising a number to a power i.e. not appreciating that 3x and x 3 are 
different (Sawyer, 1964; MacGregor and Stacey, 1994). Elsewhere, the difficulty that students have 
with rounding is reminiscent of Brown's (1981b) observation that children struggle to add one to a 
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meter reading of 06399, whilst the difficulty of converting units of area and volume, also observed by 
Tariq (2008), is clearly related to the difficulty, reported by Hart (1981b), that children have when 
asked to decide how many small cubes would fit in a larger one. 

Kuchermann (1981) and MacGregor and Stacey (1997) have both reported student difficulties with the 

different meanings that can be assigned to letters and symbols. This offers some explanation of Maths 

for Science students' problems with units and also of the fact that they are less adept at simplifying 

than rearranging equations. Others e.g. Bradis et al (1999) and Schechter (2009) have observed 

students' tendency to over-simplify and cancel inappropriately, resulting in errors of the type 

2 2 

2 2 2 o x a 

(x + y) = X + y and —-= —-. Everyday language, for example that a fraction is single 

b +cx b +c 

numerator divided by a single (larger) denominator (Pimm, 1987) further adds to the confusion. 
Finally, when a student is required, say, to divide by an algebraic fraction, they need to be able to 
understand the calculation for an arithmetic fraction and then to make the connection between 
arithmetic and algebra (Sawyer 1964). 

Errors seen in Maths for Science questions on graphs and differentiation demonstrate several of the 
difficulties reported by Kerslake (1981): 'Graphical prototypes' (e.g. assuming that all graphs go 
through the origin); 'slope-height confusion' and problems with the equation of a straight line. 
McDermott et al (1987) also explore students' difficulties in making the connection between graphs 
and physical concepts, again pointing out the difficulty that students have in discriminating between 
a value on a curve, especially when that value is a height, and the gradient at that point. 

It is possible to attempt to classify the errors observed in Maths for Science questions into separate 
categories, for example, careless mistakes (Ryan and William's (2007) 'slips'); a lack of understanding 
of a method taught in the module materials (or a misremembered method from years ago); and 
deeper conceptual misunderstandings. However there is considerable blurring of the boundaries 
between the categories and there are many examples of questions where it is difficult to be certain 
whether the common errors observed were a result of conceptual misconceptions, faulty memory or 
careless errors (Hadjidemetriou and Williams, 2002). 

It seems likely that most of the precedence errors observed in Maths for Science were simply careless 
calculator slips, since the matching errors in units were not seen. It is unfortunate that students seem 
reluctant to check that their answers are reasonable, even though checking and estimating are taught 
in the module, and it would be helpful if students understood enough about a calculation in order to 
'press the buttons on their calculator in the correct order' (Brown, 1981a, p47). At face value, errors 
such as typing Kg instead of kg (which might not be spotted in a handwritten tutor-marked 
assignment), or typing f = e/h instead of f = E/h, might also be regarded as 'minor' and/or careless. 
However, they reveal a fundamental lack of awareness of the importance of the meaning of the 
symbols used for variables and units. 

Where students have incorrectly added two fractions, it could be concluded that their fundamental 
understanding of the nature of fractions and addition is poor. However, for most adult students, the 
addition of fractions is simply a process, learnt in the module or remembered (or misremembered!) 
from childhood. The mistake of adding numerators and denominators separately has been reported 
by many others. However, following the finding that Maths for Science students sometimes start by 
correctly expressing the fractions to a common denominator but then add numerators and 
denominators separately, the teaching text was found to be unclear on this point. Improvement in the 
teaching led to an improvement in the behaviour of the EM A question (Jordan, 2013). 

The observed errors in finding the correct standard deviation are mostly attributable to using the 
incorrect formula, which is an error in mathematical process, linked to a lack of appreciation of the 
differences between the two standard deviations. 
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Errors that appear to be linked to deeper conceptual misunderstandings include a confusion between 
repeated addition i.e. multiplication (e.g. 3x) and repeated multiplication i.e. raising to a power (e.g. 
x 3 ) and difficulties with reciprocation (Rees and Barr, 1984). Similarly, a deep-rooted 
misunderstanding of physical properties such as volume and area leads to errors in conversions from, 
for example, cm 3 to m 3 , and also to errors in working out units. A confusion between the value of a 
function at a point on a graph and its gradient leads to the common errors observed both in questions 
on graphs and questions on differentiation. 

The most common errors observed in Maths for Science questions were frequently neither related to 
the question's primary purpose nor what was expected. Errors in rearranging equations were seen in 
trigonometry questions, faulty addition of fractions was seen in a probability question, precedence 
errors were seen in a statistics question etc. It is also the case that errors were often at a lower level i.e. 
with 'easier' mathematics than expected, so not only did students have difficulty expressing a number 
to an appropriate number of significant figures, they also had difficulty in simply stating the number 
of decimal places given. Howarth and Smith (1980) identify the widespread nature of problems at a 
more fundamental level than students are prepared to admit, whilst Leopold and Edgar (2008) point 
out that many tutors assume that students have a greater fluency in basic mathematics than is 
actually the case. 

There has been much discussion of the advantages and disadvantages of teaching mathematics to 
science students in a scientific context (e.g. Tariq et al, 2005; Tariq, 2008), as Maths for Science attempts 
to do. The problem of transfer from one context to another (Britton et al, 2005) means that teaching 
mathematics in a scientific context does not necessarily help. Hoban et al (2013, p. 33) report that 
mathematical difficulties experienced by chemistry students may not be caused by problems in 
transfer to the context of chemistry, but simply to students 'not possessing the necessary 
mathematical knowledge in a mathematics context.' 

Throughout Maths for Science, two groups of errors persist at a level that remains poorly understood: 
firstly, difficulties in working out the units of a final answer; secondly, problems in giving answers to 
an appropriate precision. Whilst some reasons for these errors have been suggested, further work is 
needed in order to understand the reasons for their prevalence and persistence. The Maths for Science 
book (Jordan et al, 2013) and accompanying resources are now used as part of a larger module 
Investigative and mathematical skills in science. Here the students have their own tutors and they are 
assessed partly by tutor-marked assignments. Questions have been added into tutor-marked 
assignments in an attempt to further explore the difficulties that students have in working out units. 

New screencasts, used by both Maths for Science and Investigative and mathematical skills in science, have 
been well received and point towards the benefits of using a variety of different approaches in 
teaching mathematics to science students. Another approach, suggested nearly 80 years ago by 
Benezet (1935-36), would be to introduce more discussion with students, both to learn more about the 
reasons for their errors and to improve their understanding. Synchronous and asynchronous online 
tutorials provide a way of enabling such discussion for distance-learning students anywhere in the 
world. 

Thus, interactive computer-marked assessment questions, introduced to improve the assessment and 
teaching on a Maths for Science module have also provided additional insight into students' 
mathematical errors. Where the errors are caused by deeper misconceptions, it is not possible from 
the work reported here, to be absolutely certain as to the nature of these misconceptions. However, 
given the huge numbers of responses that have been inspected, the fact that similar errors occur in 
different questions, and the fact that similar errors have been reported elsewhere, it seems that the 
errors themselves are ubiquitous and we should not ignore them. In the Open University's distance 
learning context, online tutorials provide a means by which all students can engage in discussion of 
mathematics in order to improve their understanding. Furthermore, the use of iCMAs can free human 
markers from the drudgery of routine feedback and marking (Jordan and Mitchell, 2009), saving time 
for more important work such as discussion with their students. 
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